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ABSTRACT 



This paper presents a study of 64 undergraduate students on 
the use and evaluation of scientific information. To assess scientific claims 
as they encounter them in everyday life, students need to gather information. 
Previous research suggests that a frequent first step is to generate an 
unsubstantiated casual explanation. This process could improve the search for 
new information or introduce strategies that lead to bias and distortion. The 
effect of explaining claims on the information-gathering process is examined 
in the context of investigating ecological problems. Explaining is 
manipulated as is the presence of alternative hypotheses, a common treatment 
for the undesirable effects of explanation. Results indicate that explaining 
a claim shifts the search away from covariational data that establishes the 
existence of a relationship and toward the gathering of noncovariational 
information about underlying mechanisms. This shift is partially prevented by 
the presence of alternatives. It is suggested that this shift is undesirable. 
Educational implications are discussed. (Contains 19 references.) 
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Abstract 



To assess scientific claims as they encounter them in everyday life, students need to gather 
information. Previous research suggests that a frequent first step is generating an 
unsubstantiated causal explanation. This process could improve the search for new 
information, or introduce strategies that lead to bias and distortion. The effect of explaining 
claims on the information-gathering process is examined in the context of investigating 
ecological problems. Explaining is manipulated, as is the presence of alternative hypotheses, a 
common treatment for the undesirable effects of explanation. Results show that explaimng a 
claim shifts the search away from covariational data that establishes the existence of a 
relationship, and toward the gathering of noncovariational information about underlying 
mechanisms. This shift is partially prevented by the presence of alternatives. It is argued that 
the shift is undesirable, and educational implications are discussed. 




Recent assessments of science education 
emphasize the importance of science literacy, 
i.e., the ability of students to use and evaluate 
scientific information in their everyday lives 
with confidence (American Association for the 
Advancement of Science, 1993; National 
Research Council, 1995). This can be quite a 
challenge; popular sources of scientific 
information often fail to provide the details 
needed to conduct a critical analysis. Reporting 
may only include the scientists’ conclusions, 
without explaining the data and processes that 
led them to make these claims (Brem, Weems & 
Russell, in preparation). Thus, if students are 
going to critically evaluate and use scientific 
information, they may have to fill the gaps for 
themselves. 

Brem and Rips (1995; in press) 
demonstrated that participants introduce 
unsubstantiated narrative explanations into 
arguments when evidence is lacking. Put 
simply, people tell stories about what might 
happen when data concerning what actually 
happens are missing. Given the dearth of 
information that may be available to students as 
they encounter science in their everyday lives, it 



is likely that they will engage in these speculative 
stories. 

And an explanation might not be a bad 
place to start. Ideally, we would like students to 
use the explaining process to figure out what 
they know and don’t know, and then seek out 
whatever additional information they require to 
make a decision. Explanation has been shown 
to improve comprehension, transfer, and skill 
performance (c-g., PaUncsar & Brown, 1984; 
Chi, deLeeuw, Chiu & LaVancher, 1994; Smith 
& Goodman, 1984), and increase sensitivity to 
patterns in data (e-g«, Wright & Murphy, 1986; 
Jennings, Amabile, & Ross, 1982). However, 
the body of literature touting the advantages of 
explanation is matched by an equally large body 
warning of pitfalls. Explanation has been 
shown to lead to bias and distortion in the 
search for and interpretation of evidence (e.g.. 
Chapman & Chapman, 1967, 1969), and 
overconfidence (e.g., Ross, Lepper & Hubbard, 
1975; Koehler, 1991). Therefore, it is an open 
question as to whether explaining will have a 
desirable or undesirable effect in the popular 
media environment, motivating this study. 
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There are a number of ways explaining 
could help or impede the ability to seek 
information and critically evaluate scientific 
claims; I consider only two of those here. 

Information Goals 

First, explaining may affect our goals in seeking 
information. For example, suppose we hear a 
story on a headline news channel that the 
number of red-crested hawks (a fictional 
species) in California is plummeting, and 
scientists are blaming the drop on the timber 
industry. Thus, the scientists are making the 
claim: 

Redwood harvesting is causing a decline in 

the population of red-crested hawks. 

As concerned citizens, we want to know 
whether this is claim is well-backed by evidence, 
but the broadcast doesn’t deliver the goods. 
What can they do? We can identify two rough 
categories of goals: Bstablishing existence and 
establishing a causal story. If we wish to establish 
existence, we would attempt to confirm that 
redwood harvesting reliably causes a decrease in 
the number of hawks, and that harvesting is not 
confounded with other factors. If we wish to 
establish a causal story, we can attempt to 
determine why there is a link between 
harvesting and hawk decline. For example, 
harvesting may be robbing the hawks of nesting 
sites, or the equipment used in logging may be 
producing toxic levels of pollutants. 

While establishing a mechanism is 
scientifically important, until we determine that 
there is a link between harvesting and the 
decline, such a search may be a waste of time 
and effort. Also, if we immediately jump to 
testing a particular mechanism — say, a lack of 
nesting sites — and there turns out to be no 
evidence for this particular causal path, we have 
to start again. We may have ruled out one way 
of linking harvesting to the decline, but weVe 
not ruled out the possibility of a link. Worse 
yet, given the persuasive properties of 
explanations, it could result in an unwarranted 
attachment to a false assertion. Telling 



ourselves why harvesting should lead to a decline 
may convince use that it does. Of course, 
establishing existence without a mechanism — 
mere association — ^is also inadequate. But the 
most efficient course is to make sure there is 
something worth explaining before going 
further. 

Because explanations focus on underlying 
mechanisms, I predict that explaining will 
increase the number of questions serving Story 
goals as students fail to attend to Existence 
goals. 

Information Typ e 

Second, explaining may affect the kind of 
information students seek. Both Story and 
Existence goals can be served by collecting the 
same sorts of information. In other research 
the focus has been on the distinction between 
covariational and noncovariational information 
(e.g., Ahn, KaHsh, Medin & Gelman, 1995; 
White, 1995; LaUjee et al., 1984). 

Covariational information refers to data 
concerning the presence or absence of an effect 
in the presence or absence of the proposed 
cause. For example, if the claim is that redwood 
harvesting is causing a decline in the hawk 
population, we could test this claim by asking 
the following questions: 

/. Is the population declining in places 
where harvesting is occurring (cause 
present! present) 

2. Is the population declining in places 

where no harvesting is occurring? (cause 
absent! present) 

3. Is the population stable in places where 

harvesting is occurring? (cause 

present! absent) 

4. Is the population stable in places where 

no harvesting is occurring? (cause 

absent! effect absent) 

These questions represent the four cells of 
the covariation matrix created by crossing the 
presence or absence of the proposed cause with 
the presence or absence of the target effect. 
The strongest evidence in favor of the claim 
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would be ‘‘Yes’’ to questions 1 and 4, and “No” 
to questions 2 and 3. We could also test the 
pollution mechanism in the same way: 

/. Is the population declining in places 
where pollution (of the sort harvesting 
creates) is high? (cause present! 
present) 

2. Is the population declining in places 
where pollution is low? (cause 
absent! present) 

3. Is the population stable in places where 
pollution is high? (cause present! effect 
absent) 

4. Is the population stable in places where 
pollutions is low? (cause absent! effect 
absent) 

Thus, both Existence and Story goals can 
served by collecting covariational data. 
Covariational data is often thought to be the 
strongest (Kuhn 1993); even when certain cells 
are missing, collecting covariational information 
can be informative (Klayman & Ha, 1987). 

However, researchers have found that 
people often show a bias for noncovariational 
information when making and evaluating causal 
attributions (Ahn et al., 1995; Lalljee et al., 1984; 
White, 1995). For example, we might ask “Do 
the hawks nest in dwarf redwoods?” or “Does 
harvesting produce pollution?.” These 

questions are noncovariational, in that they do 
not equate the process of making causal 
attributions with the process of matching cause 
and effect. Previous studies have equated 
noncovariational information with 

“mechanistic” information, or “hypothesis- 
testing,” making it roughly equivalent to Story 
goals. I would argue, however, that either sort 
of goal may be served by collecting 

covariational information. Likewise, although 
perhaps it seems redundant, one could just as 
easily ask “Does harvesting hurt the hawks?” 
and thus ask a noncovariational question about 
an Existence goal. If students tend not to do 
this, that is an empirical finding rather than a 
theoretical necessity. 



The distinction between covariational and 
noncovariational questions is important because 
it indicates the questioners’ focus and their 
approach to the problem of gathering evidence. 
The difference between them points to the 
questioners’ representation of the problem. 
When questioners phrase their requests as 
covariational questions, they are thinking not 
only in terms of what information they want, 
but explicidy considering relevant comparisons 
and dimensions. Noncovariational questions 
present a less well-defined request. While this 
does not rule out the possibility that the 
questioner is actually thinking in terms of the 
covariational data necessary to obtain this 
information, if there are systematic shifts from 
covariational to noncovariational questions, this 
may reflect a shift in problem representation. 

From a pedagogical standpoint, we would 
like students to approach the information 
gathering process in a way that maximizes thek 
opportunities for getting what they need, and a 
covariational approach is preferable to a 
noncovariational one in this respect. The less 
specific a question, the more ways it can be 
interpreted. “Does harvesting lead to 
pollution?” wiU no doubt get very different 
answers from conservationists and loggers; thus 
a more specific question that specifies what 
comparisons we would Uke to see is more 
deskable. Of course, the process of specifying a 
test is far more complicated than simply setting 
up appropriate dimensions of comparison — 
defining “pollution” and what constitutes high 
and low levels of pollution and harvesting is 
open to debate. But, generally speaking, the 
more specific students can be, the less 
vulnerable they are. 

Although the issue of covariation can be 
theoretically separated from the issue of goals, 
previous studies have noted a strong 
relationship between noncovariational 
information and Story goals. When people 
pursue the mechanisms underlying a 
relationship, they often do so using 
noncovariational methods. If explaining could 
leads to a rise in Story goals, it may be paralleled 



by a decrease in requests for covariational 
information. 

In summary, explaining may lead to a larger 
number of noncovariational questions, as well 
as a failure to attend to Existence goals in favor 
of Story goals. Therefore, I also considered the 
effects of a weU-documented antidote to the 
problems induced by explanation — the presence 
of alternative hypotheses. Many studies show 
that considering alternatives reduces participant 
confidence (for a review, see Koehler, 1991). 
The more possibilities that exist, the less certain 
we are that any one possibility is the right one 
(Kuhn, Flaton & Weinstock, 1994; Anderson, 
Speer & New, 1984). However, previous work 
has not addressed precisely how this uncertainty 
affects a person’s search for information. Do 
they simply lose confidence in their own 
judgment? If so, their presence should not 
counteract shifts induced by explaining. If 
alternative hypotheses encourage greater 
scrutiny and more rigorous testing, their 
presence should, relative to explaining in the 
absence of alternatives, increase the number of 
covariational questions. 

Method 

Students were presented with primary claims 
about ecological problems, couched in a short 
introductory paragraph designed to simulate the 
tone of a headline news report. Students’ task 
was to formulate three questions that they 
would pose to an expert in order to assess the 
claim’s validity. It was stressed that their job 
was to evaluate the claim, not any particular 
mechanism that might underlie the claim. We 
created four conditions by manipulating 
whether the primary claim was presented alone 
or with an alternative, and whether students did 
or did not generate an explanation for the claim. 

Participants. Sixty-four undergraduate 
students participated. 

Materials . Eight items were generated, each 
describing an ecological problem. The 
problems were fictional, to assure that students 
could not bring existing knowledge to bear. 
Each item consisted of a brief introductory 
passage, a primary claim asserting the cause of 



the problem, and an alternative claim asserting 
an alternative cause. (See Table 1 for an 
example.) 

Design & Procedure. A 2X2 between- 
participants factorial design was used. The 
independent variables were Explanation, with 
two levels (Explanation vs. No Explanation), 
and Alternatives, with two levels (Alternative 
Present vs. Alternative Absent). Across 
participants, all items appeared in all conditions. 
The order of item presentation for each 
participant was randomized. 

Table 1. An example of the items used. 

Since the beginning of this decade, there has 
been a sharp decline in the population of red- 
crested hawks in California. The decline has 
been so pronounced that the red-crested hawk 
win soon qualifj^ as an endangered animal unless 
the cause of its decline can be determined and 
removed. 

The following claim regarding the decline of the 
red-crested hawk has been made: 

Unchecked harvesting of dwarf redwoods is 
causing a decline in the number of red-crested 
hawks. 

{Alternatives Present condition only): 

Others have criticized this claim, and argue that 
the harvesting of dwarf redwoods has nothing 
to do with the decline. Instead they claim that: 

A decline in the field mouse population is 
causing a decline in the number of red-crested 
hawks. 

{Explanation condition only): 

We would like you to now try to make up an 
explanation for the effect the first claim 
describes. What specific set of events might 
lead from the unchecked harvesting of dwarf 
redwoods to the decline in the number of red- 

crested hawks? 

All participants read the introductory 
passage and primary claim. In the Alternative 
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Present condition, participants also read the 
alternative claim, and this claim was included in 
every activity they completed. 

Participants in the Explanation condition 
were instructed to explain how the stated cause 
in the primary claim could lead to the problem 
described. Having explained each of the eight 
claims, they were told to imagine that they had 
access to an expert, of whom they could ask any 
question, and to generate three questions that 
would help them to determine whether the 
primary claim were true. It was stressed that 
they were assessing the claim, not any particular 
underlying explanation, and an example was 
provided. 

Participants in the No Explanation 
condition read the initial passage and claim(s), 
then proceeded directly to the question 
generation task. This was designed to interfere 
with spontaneous explanations they might 
entertain. In all other respects, the instructions 
and tasks were identical to those used in the 
Explanation condition. 

Results 

A blind rater coded students’ questions with 
respect to goal and type of request (Table 2): 

(a) Information Goal: Existence or Story. 
The coder also had the category 
“Neither” available for questions 
regarding general background 
information (e.g., which company was 
conducting the harvesting, what 
harvested trees are used for). These 
responses were omitted from the 
analysis. 

(b) Information Type: Covariational or 

Noncovariational. 

The rater also categorized questions as to 
whether they addressed the primary claim, or an 
alternative (either a presented alternative or 
student-generated one). Students in the 
Alternatives Absent condition independently 
came up with the presented alternatives, and 
students in the Alternatives Present condition 
often did not address these alternatives in their 
questions, or came up with their own 
alternatives. Thus, it was possible to blind the 



rater to the Alternatives condition as well as the 
Explanation condition. 

Table 2. Examples of Coding Schemata 
Covariational Questions : 

“Are there other areas where hawks are 
disappearing where there are no redwoods?” 
(examining influence on effect of the 
presence/absence of the primary cause) 

“Does introducing mice into the environment 
increase the population of the hawk?” 

(examining influence on effect of the 
presence/absence of an alternate cause) 

Noncovariational Questions : 

“Do the hawks depend on the dwarf redwoods 
for nesting?” (a covariational version would be: 
“Is the number of hawk nesting sites decreasing 
with the decrease in dwarf redwoods?”) 

“Is their [hawks’] food source located in dwarf 
redwoods?” (a covariational version would be: 
“Is the amount of food available for hawks 
decreasing with the decrease in dwarf 
redwoods?”) 

Existence Questions : 

“Does introducing mice into the environment 
increase the population of the hawk?” 
(covariational) 

“How was this conclusion arrived upon?” 
(noncovariational) 

Story Questions 

“Does less trees mean less food?” 

(covariational) 

“Do the red-crested hawks lay eggs in dwarf 
redwoods?” (noncovariational) 

Questions addressing the primary claim 

Sixty-six percent of questions addressed the 
primary claim. Explaining had the predicted 
undesirable effect (Tables 3 and 4). It caused a 
sharp rise questions addressing a particular 
Story (F(l,60)= 9.25, p < 0.01). Students were 
caught up in exploring their own particular story 
and failed to establish whether there was any 
relationship to explain. Explaining also resulted 
in a rise in Noncovariational questions 
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(F(l,60)=4.31, p < 0.05); students were less 
likely to systematically lay out dimensions for 
testing. Thus, in the sound bite environment, 
explaining may be deleterious to reasoning. 

Presenting alternatives was only partially 
successful in overcoming these problems. It did 
not prevent the shift toward Story questions (F 
< 1), but did encourage Covariational questions 
(F(l ,60) =10.27, p < 0.01). We found no 
interaction between Explanation and 
Alternatives (Fs < 1). These alternatives were 
generally not part of the covariational 
comparison. That is, participants did not often 
suggest comparison that pit one claim against 
another, but rather the presence of the 
alternatives sharpened their tests of the primary 
claim on its own. 



Table 3 . Primary claim questions, coded 
with respect to information goals. 



% of Story 
questions 


Don’t 

Explain 


Explain 


Alternative 

Absent 


58.8% 


78.5% 


Alternative 

present 


56.8% 


75.8% 



Table 4 . Primary claim questions coded 
with respect to information type. 



%of 

noncovariational 

questions 


Don’t 

Explain 


Explain 


Alternative 

Absent 


53.9% 


64.6% 


Alternative 

Present 


44.6% 


48.7% 



Questions addressing an alternative claim 

Thirty-three percent of questions addressed 
alternatives, either presented or generated 
(Tables 5 and 6). Interestingly, students in the 
Alternative Present condition were no more 
likely to address alternatives than those in the 
Alternative Absent condition (34% vs. 32%). 
Explaining again caused an undesirable shift 
away from Existence questions (F(l,58)=7.50, p 
< 0.01), and away from Covariational questions, 
though this was only marginally significant 
(F(l,58)=3.47, p=0.07). We found no 



significant effects involving Alternatives, 
although the trend is towards presenting 
alternatives aggravating the effects of explaining. 
It may be that by addressing the alternative 
claim, it had effectively become the primary 
claim, and explaining thus had a negative effect. 



Table 5 . Alternative claim questions, coded 
with respect to information goals. 



% of Story 
questions 


Don’t 

Explain 


Explain 


Alternative 

Absent 


23.0% 


36.3% 


Alternative 

Present 


32.6% 


52.1% 


Table 6. Altemat 
with respect to in 


ive claim questions, coded 
formation type. 


%of 

noncovariational 

questions 


Don’t 

Explain 


Explain 


Alternative 

Absent 


49.6% 


55.0% 


Alternative 

Present 


47.7% 


64.4% 



Discussion 

Overall, explaining works against a balanced 
consideration of Existence and Story goals, 
encouraging a search for noncovariational 
information that can be used to postulate 
responsible mechanisms. The presence of 
alternatives may preserve the search for 
covariational information after explaining for 
questions addressing the primary claim, but 
these searches are nevertheless directed towards 
finding a mechanism, rather than establishing 
the presence of a relationship to account for. In 
the case of questions addressing the alternative 
claim, the presentation of alternatives, if 
anything, worsened the situation. 

It is also notable that only 33% of questions 
addressed alternatives, and presenting 
alternatives does not appear to have heightened 
students’ awareness of other possibilities. This 
is consistent with previous studies showing that 
claims tend to be pursued in isolation (Mynatt, 
Doherty, Tweney & Schiavo, 1979), and that 



explaining reinforces this tendency (Koehler, 
1991). 

Overall, these results do not recommend 
using explaining as a strategy for Filling gaps 
when evidence is lacking. Searching for a 
mechanism is not an error in itself; indeed, such 
a search is necessary to move from correlation 
to causation. Therefore, a shift to questions 
that serve Story goals may not be as problematic 
as a shift away from covariational questions. If 
no mechanism is found, perhaps the claim is 
wrong. However, this reasoning depends on 
students being able to exhaust the list of 
possible mechanisms — an unlikely scenario in 
any case, and certainly not the case when 
students ask only three questions. If an 
exhaustive search is not possible, the most 
efficient approach is to establish first that a 
relationship , exists, and then to evaluate its 
causal potency. This makes the shift away from 
questions serving Existence goals troublesome. 

More investigation regarding the effects of 
alternatives is called for. When students are 
focused on the primary goal, the presence of 
alternatives may have some benefit; students are 
not simply casting about for a mechanistic story, 
but are considering how they could test 
portions of that story. But ideally students will 
explore many possibilities, and when their focus 
switches from the primary, the benefits of 
presenting alternatives are lost. Furthermore, 
while alternatives produce less certainty in their 
wording, this does not appear to translate into a 
change in their information-seeking strategies. 
Understanding why this occurs may help use to 
determine the best way to structure students’ 
explorations. 

The motivation to look for explanations, 
even unsubstantiated explanations, has been 
well-documented as a common and 
spontaneous phenomenon (Ahn et al., 1995; 
Koehler, 1991; Brem & Rips, 1995; in press). 
While explanation had demonstrated potential 
as a tool in enhancing comprehension and 
problem-solving, it can also lead to less 
desirable strategies of inquiry. With greater 
understanding, we may be able to provide 



people with the tools they need to evaluate 
scientific issues accurately and confidendy. 
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